Frequency domain sample adaptive offset (SAO)

ABSTRACT

Frequency domain sample adaptive offset (SAO). Video processing of a first signal operates to generate a second video signal such that at least one characteristic of a first portion of video information of the first video signal is replicated in generating a second portion of video information, such that the first portion of video information and the second portion of video information undergo combination to generate the second video signal. Such use of the first video signal may involve replication and scaling of the first video information to generate the second portion of video information. One possible characteristic of the first portion of video information may correspond to an energy profile as a function of frequency. One or more portions of the first video signal may be employed to generate different respective portions of the second signal. Such video processing operations may be performed on a block by block basis.

CROSS REFERENCE TO RELATED PATENTS/PATENT APPLICATIONS ProvisionalPriority Claims

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. §119(e) to the following U.S. Provisional Patent Applicationswhich are hereby incorporated herein by reference in their entirety andmade part of the present U.S. Utility patent application for allpurposes:

1. U.S. Provisional Patent Application Ser. No. 61/597,683, entitled“Frequency domain sample adaptive offset,” (Attorney Docket No.BP24537), filed Feb. 10, 2012, pending.

2. U.S. Provisional Patent Application Ser. No. 61/603,190, entitled“Frequency domain sample adaptive offset (SAO),” (Attorney Docket No.BP24537.1), filed Feb. 24, 2012, pending.

INCORPORATION BY REFERENCE

The following standards/draft standards are hereby incorporated hereinby reference in their entirety and are made part of the present U.S.Utility patent application for all purposes:

1. “High efficiency video coding (HEVC) text specification draft 6,”Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 andISO/IEC JTC1/SC29/WG11, 7th Meeting: Geneva, CH, 21-30 November, 2011,Document: JCTVC-H1003, 259 pages.

2. International Telecommunication Union, ITU-T, TELECOMMUNICATIONSTANDARDIZATION SECTOR OF ITU, H.264 (March/2010), SERIES H: AUDIOVISUALAND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services—Coding ofmoving video, Advanced video coding for generic audiovisual services,Recommendation ITU-T H.264, also alternatively referred to asInternational Telecomm ISO/IEC 14496-10—MPEG-4 Part 10, AVC (AdvancedVideo Coding), H.264/MPEG-4 Part 10 or AVC (Advanced Video Coding), ITUH.264/MPEG4-AVC, or equivalent.

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

The invention relates generally to digital video processing; and, moreparticularly, it relates to processing and operations in accordance withsuch digital video processing.

2. Description of Related Art

Communication systems that operate to communicate digital media (e.g.,images, video, data, etc.) have been under continual development formany years. With respect to such communication systems employing someform of video data, a number of digital images are output or displayedat some frame rate (e.g., frames per second) to effectuate a videosignal suitable for output and consumption. Within many suchcommunication systems operating using video data, there can be atrade-off between throughput (e.g., number of image frames that may betransmitted from a first location to a second location) and video and/orimage quality of the signal eventually to be output or displayed. Thepresent art does not adequately or acceptably provide a means by whichvideo data may be transmitted from a first location to a second locationin accordance with providing an adequate or acceptable video and/orimage quality, ensuring a relatively low amount of overhead associatedwith the communications, relatively low complexity of the communicationdevices at respective ends of communication links, etc.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 and FIG. 2 illustrate various embodiments of communicationsystems.

FIG. 3A illustrates an embodiment of a computer.

FIG. 3B illustrates an embodiment of a laptop computer.

FIG. 3C illustrates an embodiment of a high definition (HD) television.

FIG. 3D illustrates an embodiment of a standard definition (SD)television.

FIG. 3E illustrates an embodiment of a handheld media unit.

FIG. 3F illustrates an embodiment of a set top box (STB).

FIG. 3G illustrates an embodiment of a digital video disc (DVD) player.

FIG. 3H illustrates an embodiment of a generic digital image and/orvideo processing device.

FIG. 4, FIG. 5, and FIG. 6 are diagrams illustrating various embodimentsof video encoding architectures.

FIG. 7 is a diagram illustrating an embodiment of intra-predictionprocessing.

FIG. 8 is a diagram illustrating an embodiment of inter-predictionprocessing.

FIG. 9 and FIG. 10 are diagrams illustrating various embodiments ofvideo decoding architectures.

FIG. 11 illustrates an embodiment of a video encoding architecture (withsample adaptive offset (SAO) implemented after de-blocking).

FIG. 12 illustrates an embodiment of a video encoding architecture (witha frequency-domain sample adaptive offset (FSAO) implemented afterinverse quantization).

FIG. 13 illustrates an embodiment of principles of spectral bandreplication (SBR)—transposition (a) and reconstruction (b).

FIG. 14, FIG. 15, FIG. 16, and FIG. 17 illustrate various embodiments ofspectral band replication (SBR).

FIG. 18, FIG. 19, FIG. 20A, and FIG. 20B illustrate various embodimentsof methods for performing video processing (e.g., within one or moredevices).

DETAILED DESCRIPTION OF THE INVENTION

Within many devices that use digital media such as digital video,respective images thereof, being digital in nature, are representedusing pixels. Within certain communication systems, digital media can betransmitted from a first location to a second location at which suchmedia can be output or displayed. The goal of digital communicationssystems, including those that operate to communicate digital video, isto transmit digital data from one location, or subsystem, to anothereither error free or with an acceptably low error rate. As shown in FIG.1, data may be transmitted over a variety of communications channels ina wide variety of communication systems: magnetic media, wired,wireless, fiber, copper, and/or other types of media as well.

FIG. 1 and FIG. 2 are diagrams illustrate various embodiments ofcommunication systems, 100 and 200, respectively.

Referring to FIG. 1, this embodiment of a communication system 100 is acommunication channel 199 that communicatively couples a communicationdevice 110 (including a transmitter 112 having an encoder 114 andincluding a receiver 116 having a decoder 118) situated at one end ofthe communication channel 199 to another communication device 120(including a transmitter 126 having an encoder 128 and including areceiver 122 having a decoder 124) at the other end of the communicationchannel 199. In some embodiments, either of the communication devices110 and 120 may only include a transmitter or a receiver. There areseveral different types of media by which the communication channel 199may be implemented (e.g., a satellite communication channel 130 usingsatellite dishes 132 and 134, a wireless communication channel 140 usingtowers 142 and 144 and/or local antennae 152 and 154, a wiredcommunication channel 150, and/or a fiber-optic communication channel160 using electrical to optical (E/O) interface 162 and optical toelectrical (O/E) interface 164)). In addition, more than one type ofmedia may be implemented and interfaced together thereby forming thecommunication channel 199.

It is noted that such communication devices 110 and/or 120 may bestationary or mobile without departing from the scope and spirit of theinvention. For example, either one or both of the communication devices110 and 120 may be implemented in a fixed location or may be a mobilecommunication device with capability to associate with and/orcommunicate with more than one network access point (e.g., differentrespective access points (APs) in the context of a mobile communicationsystem including one or more wireless local area networks (WLANs),different respective satellites in the context of a mobile communicationsystem including one or more satellite, or generally, differentrespective network access points in the context of a mobilecommunication system including one or more network access points bywhich communications may be effectuated with communication devices 110and/or 120.

To reduce transmission errors that may undesirably be incurred within acommunication system, error correction and channel coding schemes areoften employed. Generally, these error correction and channel codingschemes involve the use of an encoder at the transmitter end of thecommunication channel 199 and a decoder at the receiver end of thecommunication channel 199.

Any of various types of ECC codes described can be employed within anysuch desired communication system (e.g., including those variationsdescribed with respect to FIG. 1), any information storage device (e.g.,hard disk drives (HDDs), network information storage devices and/orservers, etc.) or any application in which information encoding and/ordecoding is desired.

Generally speaking, when considering a communication system in whichvideo data is communicated from one location, or subsystem, to another,video data encoding may generally be viewed as being performed at atransmitting end of the communication channel 199, and video datadecoding may generally be viewed as being performed at a receiving endof the communication channel 199.

Also, while the embodiment of this diagram shows bi-directionalcommunication being capable between the communication devices 110 and120, it is of course noted that, in some embodiments, the communicationdevice 110 may include only video data encoding capability, and thecommunication device 120 may include only video data decodingcapability, or vice versa (e.g., in a uni-directional communicationembodiment such as in accordance with a video broadcast embodiment).

Referring to the communication system 200 of FIG. 2, at a transmittingend of a communication channel 299, information bits 201 (e.g.,corresponding particularly to video data in one embodiment) are providedto a transmitter 297 that is operable to perform encoding of theseinformation bits 201 using an encoder and symbol mapper 220 (which maybe viewed as being distinct functional blocks 222 and 224, respectively)thereby generating a sequence of discrete-valued modulation symbols 203that is provided to a transmit driver 230 that uses a DAC (Digital toAnalog Converter) 232 to generate a continuous-time transmit signal 204and a transmit filter 234 to generate a filtered, continuous-timetransmit signal 205 that substantially comports with the communicationchannel 299. At a receiving end of the communication channel 299,continuous-time receive signal 206 is provided to an AFE (Analog FrontEnd) 260 that includes a receive filter 262 (that generates a filtered,continuous-time receive signal 207) and an ADC (Analog to DigitalConverter) 264 (that generates discrete-time receive signals 208). Ametric generator 270 calculates metrics 209 (e.g., on either a symboland/or bit basis) that are employed by a decoder 280 to make bestestimates of the discrete-valued modulation symbols and information bitsencoded therein 210.

Within each of the transmitter 297 and the receiver 298, any desiredintegration of various components, blocks, functional blocks,circuitries, etc. Therein may be implemented. For example, this diagramshows a processing module 280 a as including the encoder and symbolmapper 220 and all associated, corresponding components therein, and aprocessing module 280 is shown as including the metric generator 270 andthe decoder 280 and all associated, corresponding components therein.Such processing modules 280 a and 280 b may be respective integratedcircuits. Of course, other boundaries and groupings may alternatively beperformed without departing from the scope and spirit of the invention.For example, all components within the transmitter 297 may be includedwithin a first processing module or integrated circuit, and allcomponents within the receiver 298 may be included within a secondprocessing module or integrated circuit. Alternatively, any othercombination of components within each of the transmitter 297 and thereceiver 298 may be made in other embodiments.

As with the previous embodiment, such a communication system 200 may beemployed for the communication of video data is communicated from onelocation, or subsystem, to another (e.g., from transmitter 297 to thereceiver 298 via the communication channel 299).

Digital image and/or video processing of digital images and/or media(including the respective images within a digital video signal) may beperformed by any of the various devices depicted below in FIG. 3A-3H toallow a user to view such digital images and/or video. These variousdevices do not include an exhaustive list of devices in which the imageand/or video processing described herein may be effectuated, and it isnoted that any generic digital image and/or video processing device maybe implemented to perform the processing described herein withoutdeparting from the scope and spirit of the invention.

FIG. 3A illustrates an embodiment of a computer 301. The computer 301can be a desktop computer, or an enterprise storage devices such aserver, of a host computer that is attached to a storage array such as aredundant array of independent disks (RAID) array, storage router, edgerouter, storage switch and/or storage director. A user is able to viewstill digital images and/or video (e.g., a sequence of digital images)using the computer 301. Oftentimes, various image and/or video viewingprograms and/or media player programs are included on a computer 301 toallow a user to view such images (including video).

FIG. 3B illustrates an embodiment of a laptop computer 302. Such alaptop computer 302 may be found and used in any of a wide variety ofcontexts. In recent years, with the ever-increasing processingcapability and functionality found within laptop computers, they arebeing employed in many instances where previously higher-end and morecapable desktop computers would be used. As with the computer 301, thelaptop computer 302 may include various image viewing programs and/ormedia player programs to allow a user to view such images (includingvideo).

FIG. 3C illustrates an embodiment of a high definition (HD) television303. Many HD televisions 303 include an integrated tuner to allow thereceipt, processing, and decoding of media content (e.g., televisionbroadcast signals) thereon. Alternatively, sometimes an HD television303 receives media content from another source such as a digital videodisc (DVD) player, set top box (STB) that receives, processes, anddecodes a cable and/or satellite television broadcast signal. Regardlessof the particular implementation, the HD television 303 may beimplemented to perform image and/or video processing as describedherein. Generally speaking, an HD television 303 has capability todisplay HD media content and oftentimes is implemented having a 16:9widescreen aspect ratio.

FIG. 3D illustrates an embodiment of a standard definition (SD)television 304. Of course, an SD television 304 is somewhat analogous toan HD television 303, with at least one difference being that the SDtelevision 304 does not include capability to display HD media content,and an SD television 304 oftentimes is implemented having a 4:3 fullscreen aspect ratio. Nonetheless, even an SD television 304 may beimplemented to perform image and/or video processing as describedherein.

FIG. 3E illustrates an embodiment of a handheld media unit 305. Ahandheld media unit 305 may operate to provide general storage orstorage of image/video content information such as joint photographicexperts group (JPEG) files, tagged image file format (TIFF), bitmap,motion picture experts group (MPEG) files, Windows Media (WMA/WMV)files, other types of video content such as MPEG4 files, etc. forplayback to a user, and/or any other type of information that may bestored in a digital format. Historically, such handheld media units wereprimarily employed for storage and playback of audio media; however,such a handheld media unit 305 may be employed for storage and playbackof virtual any media (e.g., audio media, video media, photographicmedia, etc.). Moreover, such a handheld media unit 305 may also includeother functionality such as integrated communication circuitry for wiredand wireless communications. Such a handheld media unit 305 may beimplemented to perform image and/or video processing as describedherein.

FIG. 3F illustrates an embodiment of a set top box (STB) 306. Asmentioned above, sometimes a STB 306 may be implemented to receive,process, and decode a cable and/or satellite television broadcast signalto be provided to any appropriate display capable device such as SDtelevision 304 and/or HD television 303. Such an STB 306 may operateindependently or cooperatively with such a display capable device toperform image and/or video processing as described herein.

FIG. 3G illustrates an embodiment of a digital video disc (DVD) player307. Such a DVD player may be a Blu-Ray DVD player, an HD capable DVDplayer, an SD capable DVD player, an up-sampling capable DVD player(e.g., from SD to HD, etc.) without departing from the scope and spiritof the invention. The DVD player may provide a signal to any appropriatedisplay capable device such as SD television 304 and/or HD television303. The DVD player 305 may be implemented to perform image and/or videoprocessing as described herein.

FIG. 3H illustrates an embodiment of a generic digital image and/orvideo processing device 308. Again, as mentioned above, these variousdevices described above do not include an exhaustive list of devices inwhich the image and/or video processing described herein may beeffectuated, and it is noted that any generic digital image and/or videoprocessing device 308 may be implemented to perform the image and/orvideo processing described herein without departing from the scope andspirit of the invention.

FIG. 4, FIG. 5, and FIG. 6 are diagrams illustrating various embodiments400 and 500, and 600, respectively, of video encoding architectures.

Referring to embodiment 400 of FIG. 4, as may be seen with respect tothis diagram, an input video signal is received by a video encoder. Incertain embodiments, the input video signal is composed of coding units(CUs) or macro-blocks (MBs). The size of such coding units ormacro-blocks may be varied and can include a number of pixels typicallyarranged in a square shape. In one embodiment, such coding units ormacro-blocks have a size of 16×16 pixels. However, it is generally notedthat a macro-block may have any desired size such as N×N pixels, where Nis an integer. Of course, some implementations may include non-squareshaped coding units or macro-blocks, although square shaped coding unitsor macro-blocks are employed in a preferred embodiment.

The input video signal may generally be referred to as corresponding toraw frame (or picture) image data. For example, raw frame (or picture)image data may undergo processing to generate luma and chroma samples.In some embodiments, the set of luma samples in a macro-block is of oneparticular arrangement (e.g., 16×16), and set of the chroma samples isof a different particular arrangement (e.g., 8×8). In accordance withthe embodiment depicted herein, a video encoder processes such sampleson a block by block basis.

The input video signal then undergoes mode selection by which the inputvideo signal selectively undergoes intra and/or inter-predictionprocessing. Generally speaking, the input video signal undergoescompression along a compression pathway. When operating with no feedback(e.g., in accordance with neither inter-prediction norintra-prediction), the input video signal is provided via thecompression pathway to undergo transform operations (e.g., in accordancewith discrete cosine transform (DCT)). Of course, other transforms maybe employed in alternative embodiments. In this mode of operation, theinput video signal itself is that which is compressed. The compressionpathway may take advantage of the lack of high frequency sensitivity ofhuman eyes in performing the compression.

However, feedback may be employed along the compression pathway byselectively using inter- or intra-prediction video encoding. Inaccordance with a feedback or predictive mode of operation, thecompression pathway operates on a (relatively low energy) residual(e.g., a difference) resulting from subtraction of a predicted value ofa current macro-block from the current macro-block. Depending upon whichform of prediction is employed in a given instance, a residual ordifference between a current macro-block and a predicted value of thatmacro-block based on at least a portion of that same frame (or picture)or on at least a portion of at least one other frame (or picture) isgenerated.

The resulting modified video signal then undergoes transform operationsalong the compression pathway. In one embodiment, a discrete cosinetransform (DCT) operates on a set of video samples (e.g., luma, chroma,residual, etc.) to compute respective coefficient values for each of apredetermined number of basis patterns. For example, one embodimentincludes 64 basis functions (e.g., such as for an 8×8 sample). Generallyspeaking, different embodiments may employ different numbers of basisfunctions (e.g., different transforms). Any combination of thoserespective basis functions, including appropriate and selectiveweighting thereof, may be used to represent a given set of videosamples. Additional details related to various ways of performingtransform operations are described in the technical literatureassociated with video encoding including those standards/draft standardsthat have been incorporated by reference as indicated above. The outputfrom the transform processing includes such respective coefficientvalues. This output is provided to a quantizer.

Generally, most image blocks will typically yield coefficients (e.g.,DCT coefficients in an embodiment operating in accordance with discretecosine transform (DCT)) such that the most relevant DCT coefficients areof lower frequencies. Because of this and of the human eyes' relativelypoor sensitivity to high frequency visual effects, a quantizer may beoperable to convert most of the less relevant coefficients to a value ofzero. That is to say, those coefficients whose relative contribution isbelow some predetermined value (e.g., some threshold) may be eliminatedin accordance with the quantization process. A quantizer may also beoperable to convert the significant coefficients into values that can becoded more efficiently than those that result from the transformprocess. For example, the quantization process may operate by dividingeach respective coefficient by an integer value and discarding anyremainder. Such a process, when operating on typical coding units ormacro-blocks, typically yields a relatively low number of non-zerocoefficients which are then delivered to an entropy encoder for losslessencoding and for use in accordance with a feedback path which may selectintra-prediction and/or inter-prediction processing in accordance withvideo encoding.

An entropy encoder operates in accordance with a lossless compressionencoding process. In comparison, the quantization operations aregenerally lossy. The entropy encoding process operates on thecoefficients provided from the quantization process. Those coefficientsmay represent various characteristics (e.g., luma, chroma, residual,etc.). Various types of encoding may be employed by an entropy encoder.For example, context-adaptive binary arithmetic coding (CABAC) and/orcontext-adaptive variable-length coding (CAVLC) may be performed by theentropy encoder. For example, in accordance with at least one part of anentropy coding scheme, the data is converted to a (run, level) pairing(e.g., data 14, 3, 0, 4, 0, 0, −3 would be converted to the respective(run, level) pairs of (0, 14), (0, 3), (1, 4), (2,−3)). In advance, atable may be prepared that assigns variable length codes for valuepairs, such that relatively shorter length codes are assigned torelatively common value pairs, and relatively longer length codes areassigned for relatively less common value pairs.

As the reader will understand, the operations of inverse quantizationand inverse transform correspond to those of quantization and transform,respectively. For example, in an embodiment in which a DCT is employedwithin the transform operations, then an inverse DCT (IDCT) is thatemployed within the inverse transform operations.

A picture buffer, alternatively referred to as a digital picture bufferor a DPB, receives the signal from the IDCT module; the picture bufferis operative to store the current frame (or picture) and/or one or moreother frames (or pictures) such as may be used in accordance withintra-prediction and/or inter-prediction operations as may be performedin accordance with video encoding. It is noted that in accordance withintra-prediction, a relatively small amount of storage may besufficient, in that, it may not be necessary to store the current frame(or picture) or any other frame (or picture) within the frame (orpicture) sequence. Such stored information may be employed forperforming motion compensation and/or motion estimation in the case ofperforming inter-prediction in accordance with video encoding.

In one possible embodiment, for motion estimation, a respective set ofluma samples (e.g., 16×16) from a current frame (or picture) arecompared to respective buffered counterparts in other frames (orpictures) within the frame (or picture) sequence (e.g., in accordancewith inter-prediction). In one possible implementation, a closestmatching area is located (e.g., prediction reference) and a vectoroffset (e.g., motion vector) is produced. In a single frame (orpicture), a number of motion vectors may be found and not all willnecessarily point in the same direction. One or more operations asperformed in accordance with motion estimation are operative to generateone or more motion vectors.

Motion compensation is operative to employ one or more motion vectors asmay be generated in accordance with motion estimation. A predictionreference set of samples is identified and delivered for subtractionfrom the original input video signal in an effort hopefully to yield arelatively (e.g., ideally, much) lower energy residual. If suchoperations do not result in a yielded lower energy residual, motioncompensation need not necessarily be performed and the transformoperations may merely operate on the original input video signal insteadof on a residual (e.g., in accordance with an operational mode in whichthe input video signal is provided straight through to the transformoperation, such that neither intra-prediction nor inter-prediction areperformed), or intra-prediction may be utilized and transform operationsperformed on the residual resulting from intra-prediction. Also, if themotion estimation and/or motion compensation operations are successful,the motion vector may also be sent to the entropy encoder along with thecorresponding residual's coefficients for use in undergoing losslessentropy encoding.

The output from the overall video encoding operation is an output bitstream. It is noted that such an output bit stream may of course undergocertain processing in accordance with generating a continuous timesignal which may be transmitted via a communication channel. Forexample, certain embodiments operate within wireless communicationsystems. In such an instance, an output bitstream may undergoappropriate digital to analog conversion, frequency conversion, scaling,filtering, modulation, symbol mapping, and/or any other operationswithin a wireless communication device that operate to generate acontinuous time signal capable of being transmitted via a communicationchannel, etc.

Referring to embodiment 500 of FIG. 5, as may be seen with respect tothis diagram, an input video signal is received by a video encoder. Incertain embodiments, the input video signal is composed of coding unitsor macro-blocks (and/or may be partitioned into coding units (CUs)). Thesize of such coding units or macro-blocks may be varied and can includea number of pixels typically arranged in a square shape. In oneembodiment, such coding units or macro-blocks have a size of 16×16pixels. However, it is generally noted that a macro-block may have anydesired size such as N×N pixels, where N is an integer. Of course, someimplementations may include non-square shaped coding units ormacro-blocks, although square shaped coding units or macro-blocks areemployed in a preferred embodiment.

The input video signal may generally be referred to as corresponding toraw frame (or picture) image data. For example, raw frame (or picture)image data may undergo processing to generate luma and chroma samples.In some embodiments, the set of luma samples in a macro-block is of oneparticular arrangement (e.g., 16×16), and set of the chroma samples isof a different particular arrangement (e.g., 8×8). In accordance withthe embodiment depicted herein, a video encoder processes such sampleson a block by block basis.

The input video signal then undergoes mode selection by which the inputvideo signal selectively undergoes intra and/or inter-predictionprocessing. Generally speaking, the input video signal undergoescompression along a compression pathway. When operating with no feedback(e.g., in accordance with neither inter-prediction norintra-prediction), the input video signal is provided via thecompression pathway to undergo transform operations (e.g., in accordancewith discrete cosine transform (DCT)). Of course, other transforms maybe employed in alternative embodiments. In this mode of operation, theinput video signal itself is that which is compressed. The compressionpathway may take advantage of the lack of high frequency sensitivity ofhuman eyes in performing the compression.

However, feedback may be employed along the compression pathway byselectively using inter- or intra-prediction video encoding. Inaccordance with a feedback or predictive mode of operation, thecompression pathway operates on a (relatively low energy) residual(e.g., a difference) resulting from subtraction of a predicted value ofa current macro-block from the current macro-block. Depending upon whichform of prediction is employed in a given instance, a residual ordifference between a current macro-block and a predicted value of thatmacro-block based on at least a portion of that same frame (or picture)or on at least a portion of at least one other frame (or picture) isgenerated.

The resulting modified video signal then undergoes transform operationsalong the compression pathway. In one embodiment, a discrete cosinetransform (DCT) operates on a set of video samples (e.g., luma, chroma,residual, etc.) to compute respective coefficient values for each of apredetermined number of basis patterns. For example, one embodimentincludes 64 basis functions (e.g., such as for an 8×8 sample). Generallyspeaking, different embodiments may employ different numbers of basisfunctions (e.g., different transforms). Any combination of thoserespective basis functions, including appropriate and selectiveweighting thereof, may be used to represent a given set of videosamples. Additional details related to various ways of performingtransform operations are described in the technical literatureassociated with video encoding including those standards/draft standardsthat have been incorporated by reference as indicated above. The outputfrom the transform processing includes such respective coefficientvalues. This output is provided to a quantizer.

Generally, most image blocks will typically yield coefficients (e.g.,DCT coefficients in an embodiment operating in accordance with discretecosine transform (DCT)) such that the most relevant DCT coefficients areof lower frequencies. Because of this and of the human eyes' relativelypoor sensitivity to high frequency visual effects, a quantizer may beoperable to convert most of the less relevant coefficients to a value ofzero. That is to say, those coefficients whose relative contribution isbelow some predetermined value (e.g., some threshold) may be eliminatedin accordance with the quantization process. A quantizer may also beoperable to convert the significant coefficients into values that can becoded more efficiently than those that result from the transformprocess. For example, the quantization process may operate by dividingeach respective coefficient by an integer value and discarding anyremainder. Such a process, when operating on typical coding units ormacro-blocks, typically yields a relatively low number of non-zerocoefficients which are then delivered to an entropy encoder for losslessencoding and for use in accordance with a feedback path which may selectintra-prediction and/or inter-prediction processing in accordance withvideo encoding.

An entropy encoder operates in accordance with a lossless compressionencoding process. In comparison, the quantization operations aregenerally lossy. The entropy encoding process operates on thecoefficients provided from the quantization process. Those coefficientsmay represent various characteristics (e.g., luma, chroma, residual,etc.). Various types of encoding may be employed by an entropy encoder.For example, context-adaptive binary arithmetic coding (CABAC) and/orcontext-adaptive variable-length coding (CAVLC) may be performed by theentropy encoder. For example, in accordance with at least one part of anentropy coding scheme, the data is converted to a (run, level) pairing(e.g., data 14, 3, 0, 4, 0, 0, −3 would be converted to the respective(run, level) pairs of (0, 14), (0, 3), (1, 4), (2,−3)). In advance, atable may be prepared that assigns variable length codes for valuepairs, such that relatively shorter length codes are assigned torelatively common value pairs, and relatively longer length codes areassigned for relatively less common value pairs.

As the reader will understand, the operations of inverse quantizationand inverse transform correspond to those of quantization and transform,respectively. For example, in an embodiment in which a DCT is employedwithin the transform operations, then an inverse DCT (IDCT) is thatemployed within the inverse transform operations.

An adaptive loop filter (ALF) is implemented to process the output fromthe inverse transform block. Such an adaptive loop filter (ALF) isapplied to the decoded picture before it is stored in a picture buffer(sometimes referred to as a DPB, digital picture buffer). The adaptiveloop filter (ALF) is implemented to reduce coding noise of the decodedpicture, and the filtering thereof may be selectively applied on a sliceby slice basis, respectively, for luminance and chrominance whether ornot the adaptive loop filter (ALF) is applied either at slice level orat block level. Two-dimensional 2-D finite impulse response (FIR)filtering may be used in application of the adaptive loop filter (ALF).The coefficients of the filters may be designed slice by slice at theencoder, and such information is then signaled to the decoder (e.g.,signaled from a transmitter communication device including a videoencoder [alternatively referred to as encoder] to a receivercommunication device including a video decoder [alternatively referredto as decoder]).

One embodiment operates by generating the coefficients in accordancewith Wiener filtering design. In addition, it may be applied on a blockby block based at the encoder whether the filtering is performed andsuch a decision is then signaled to the decoder (e.g., signaled from atransmitter communication device including a video encoder[alternatively referred to as encoder] to a receiver communicationdevice including a video decoder [alternatively referred to as decoder])based on quadtree structure, where the block size is decided accordingto the rate-distortion optimization. It is noted that the implementationof using such 2-D filtering may introduce a degree of complexity inaccordance with both encoding and decoding. For example, by using 2-Dfiltering in accordance and implementation of an adaptive loop filter(ALF), there may be some increasing complexity within encoderimplemented within the transmitter communication device as well aswithin a decoder implemented within a receiver communication device.

In certain optional embodiments, the output from the de-blocking filteris provided to one or more other in-loop filters (e.g., implemented inaccordance with adaptive loop filter (ALF), sample adaptive offset (SAO)filter, and/or any other filter type) implemented to process the outputfrom the inverse transform block. For example, such an ALF is applied tothe decoded picture before it is stored in a picture buffer (again,sometimes alternatively referred to as a DPB, digital picture buffer).Such an ALF is implemented to reduce coding noise of the decodedpicture, and the filtering thereof may be selectively applied on a sliceby slice basis, respectively, for luminance and chrominance whether ornot such an ALF is applied either at slice level or at block level.Two-dimensional 2-D finite impulse response (FIR) filtering may be usedin application of such an ALF. The coefficients of the filters may bedesigned slice by slice at the encoder, and such information is thensignaled to the decoder (e.g., signaled from a transmitter communicationdevice including a video encoder [alternatively referred to as encoder]to a receiver communication device including a video decoder[alternatively referred to as decoder]).

One embodiment is operative to generate the coefficients in accordancewith Wiener filtering design. In addition, it may be applied on a blockby block based at the encoder whether the filtering is performed andsuch a decision is then signaled to the decoder (e.g., signaled from atransmitter communication device including a video encoder[alternatively referred to as encoder] to a receiver communicationdevice including a video decoder [alternatively referred to as decoder])based on quadtree structure, where the block size is decided accordingto the rate-distortion optimization. It is noted that the implementationof using such 2-D filtering may introduce a degree of complexity inaccordance with both encoding and decoding. For example, by using 2-Dfiltering in accordance and implementation of an ALF, there may be someincreasing complexity within encoder implemented within the transmittercommunication device as well as within a decoder implemented within areceiver communication device.

As mentioned with respect to other embodiments, the use of an ALF canprovide any of a number of improvements in accordance with such videoprocessing, including an improvement on the objective quality measure bythe peak to signal noise ratio (PSNR) that comes from performing randomquantization noise removal. In addition, the subjective quality of asubsequently encoded video signal may be achieved from illuminationcompensation, which may be introduced in accordance with performingoffset processing and scaling processing (e.g., in accordance withapplying a gain) in accordance with ALF processing.

With respect to one type of an in-loop filter, the use of an adaptiveloop filter (ALF) can provide any of a number of improvements inaccordance with such video processing, including an improvement on theobjective quality measure by the peak to signal noise ratio (PSNR) thatcomes from performing random quantization noise removal. In addition,the subjective quality of a subsequently encoded video signal may beachieved from illumination compensation, which may be introduced inaccordance with performing offset processing and scaling processing(e.g., in accordance with applying a gain) in accordance with adaptiveloop filter (ALF) processing.

Receiving the signal output from the ALF is a picture buffer,alternatively referred to as a digital picture buffer or a DPB; thepicture buffer is operative to store the current frame (or picture)and/or one or more other frames (or pictures) such as may be used inaccordance with intra-prediction and/or inter-prediction operations asmay be performed in accordance with video encoding. It is noted that inaccordance with intra-prediction, a relatively small amount of storagemay be sufficient, in that, it may not be necessary to store the currentframe (or picture) or any other frame (or picture) within the frame (orpicture) sequence. Such stored information may be employed forperforming motion compensation and/or motion estimation in the case ofperforming inter-prediction in accordance with video encoding.

In one possible embodiment, for motion estimation, a respective set ofluma samples (e.g., 16×16) from a current frame (or picture) arecompared to respective buffered counterparts in other frames (orpictures) within the frame (or picture) sequence (e.g., in accordancewith inter-prediction). In one possible implementation, a closestmatching area is located (e.g., prediction reference) and a vectoroffset (e.g., motion vector) is produced. In a single frame (orpicture), a number of motion vectors may be found and not all willnecessarily point in the same direction. One or more operations asperformed in accordance with motion estimation are operative to generateone or more motion vectors.

Motion compensation is operative to employ one or more motion vectors asmay be generated in accordance with motion estimation. A predictionreference set of samples is identified and delivered for subtractionfrom the original input video signal in an effort hopefully to yield arelatively (e.g., ideally, much) lower energy residual. If suchoperations do not result in a yielded lower energy residual, motioncompensation need not necessarily be performed and the transformoperations may merely operate on the original input video signal insteadof on a residual (e.g., in accordance with an operational mode in whichthe input video signal is provided straight through to the transformoperation, such that neither intra-prediction nor inter-prediction areperformed), or intra-prediction may be utilized and transform operationsperformed on the residual resulting from intra-prediction. Also, if themotion estimation and/or motion compensation operations are successful,the motion vector may also be sent to the entropy encoder along with thecorresponding residual's coefficients for use in undergoing losslessentropy encoding.

The output from the overall video encoding operation is an output bitstream. It is noted that such an output bit stream may of course undergocertain processing in accordance with generating a continuous timesignal which may be transmitted via a communication channel. Forexample, certain embodiments operate within wireless communicationsystems. In such an instance, an output bitstream may undergoappropriate digital to analog conversion, frequency conversion, scaling,filtering, modulation, symbol mapping, and/or any other operationswithin a wireless communication device that operate to generate acontinuous time signal capable of being transmitted via a communicationchannel, etc.

Referring to embodiment 600 of FIG. 6, with respect to this diagramdepicting an alternative embodiment of a video encoder, such a videoencoder carries out prediction, transform, and encoding processes toproduce a compressed output bit stream. Such a video encoder may operatein accordance with and be compliant with one or more video encodingprotocols, standards, and/or recommended practices such as ISO/IEC14496-10—MPEG-4 Part 10, AVC (Advanced Video Coding), alternativelyreferred to as H.264/MPEG-4 Part 10 or AVC (Advanced Video Coding), ITUH.264/MPEG4-AVC.

It is noted that a corresponding video decoder, such as located within adevice at another end of a communication channel, is operative toperform the complementary processes of decoding, inverse transform, andreconstruction to produce a respective decoded video sequence that is(ideally) representative of the input video signal.

As may be seen with respect to this diagram, alternative arrangementsand architectures may be employed for effectuating video encoding.Generally speaking, an encoder processes an input video signal (e.g.,typically composed in units of coding units or macro-blocks, often timesbeing square in shape and including N×N pixels therein). The videoencoding determines a prediction of the current macro-block based onpreviously coded data. That previously coded data may come from thecurrent frame (or picture) itself (e.g., such as in accordance withintra-prediction) or from one or more other frames (or pictures) thathave already been coded (e.g., such as in accordance withinter-prediction). The video encoder subtracts the prediction of thecurrent macro-block to form a residual.

Generally speaking, intra-prediction is operative to employ block sizesof one or more particular sizes (e.g., 16×16, 8×8, or 4×4) to predict acurrent macro-block from surrounding, previously coded pixels within thesame frame (or picture). Generally speaking, inter-prediction isoperative to employ a range of block sizes (e.g., 16×16 down to 4×4) topredict pixels in the current frame (or picture) from regions that areselected from within one or more previously coded frames (or pictures).

With respect to the transform and quantization operations, a block ofresidual samples may undergo transformation using a particular transform(e.g., 4×4 or 8×8). One possible embodiment of such a transform operatesin accordance with discrete cosine transform (DCT). The transformoperation outputs a group of coefficients such that each respectivecoefficient corresponds to a respective weighting value of one or morebasis functions associated with a transform. After undergoingtransformation, a block of transform coefficients is quantized (e.g.,each respective coefficient may be divided by an integer value and anyassociated remainder may be discarded, or they may be multiplied by aninteger value). The quantization process is generally inherently lossy,and it can reduce the precision of the transform coefficients accordingto a quantization parameter (QP). Typically, many of the coefficientsassociated with a given macro-block are zero, and only some non-zerocoefficients remain. Generally, a relatively high QP setting isoperative to result in a greater proportion of zero-valued coefficientsand smaller magnitudes of non-zero coefficients, resulting in relativelyhigh compression (e.g., relatively lower coded bit rate) at the expenseof relatively poorly decoded image quality; a relatively low QP settingis operative to allow more non-zero coefficients to remain afterquantization and larger magnitudes of non-zero coefficients, resultingin relatively lower compression (e.g., relatively higher coded bit rate)with relatively better decoded image quality.

The video encoding process produces a number of values that are encodedto form the compressed bit stream. Examples of such values include thequantized transform coefficients, information to be employed by adecoder to re-create the appropriate prediction, information regardingthe structure of the compressed data and compression tools employedduring encoding, information regarding a complete video sequence, etc.Such values and/or parameters (e.g., syntax elements) may undergoencoding within an entropy encoder operating in accordance with CABAC,CAVLC, or some other entropy coding scheme, to produce an output bitstream that may be stored, transmitted (e.g., after undergoingappropriate processing to generate a continuous time signal thatcomports with a communication channel), etc.

In an embodiment operating using a feedback path, the output of thetransform and quantization undergoes inverse quantization and inversetransform. One or both of intra-prediction and inter-prediction may beperformed in accordance with video encoding. Also, motion compensationand/or motion estimation may be performed in accordance with such videoencoding.

The signal path output from the inverse quantization and inversetransform (e.g., IDCT) block, which is provided to the intra-predictionblock, is also provided to a de-blocking filter. The output from thede-blocking filter is provided to one or more other in-loop filters(e.g., implemented in accordance with adaptive loop filter (ALF), sampleadaptive offset (SAO) filter, and/or any other filter type) implementedto process the output from the inverse transform block. For example, inone possible embodiment, an ALF is applied to the decoded picture beforeit is stored in a picture buffer (again, sometimes alternativelyreferred to as a DPB, digital picture buffer). The ALF is implemented toreduce coding noise of the decoded picture, and the filtering thereofmay be selectively applied on a slice by slice basis, respectively, forluminance and chrominance whether or not the ALF is applied either atslice level or at block level. Two-dimensional 2-D finite impulseresponse (FIR) filtering may be used in application of the ALF. Thecoefficients of the filters may be designed slice by slice at theencoder, and such information is then signaled to the decoder (e.g.,signaled from a transmitter communication device including a videoencoder [alternatively referred to as encoder] to a receivercommunication device including a video decoder [alternatively referredto as decoder]).

One embodiment generated the coefficients in accordance with Wienerfiltering design. In addition, it may be applied on a block by blockbased at the encoder whether the filtering is performed and such adecision is then signaled to the decoder (e.g., signaled from atransmitter communication device including a video encoder[alternatively referred to as encoder] to a receiver communicationdevice including a video decoder [alternatively referred to as decoder])based on quadtree structure, where the block size is decided accordingto the rate-distortion optimization. It is noted that the implementationof using such 2-D filtering may introduce a degree of complexity inaccordance with both encoding and decoding. For example, by using 2-Dfiltering in accordance and implementation of an ALF, there may be someincreasing complexity within encoder implemented within the transmittercommunication device as well as within a decoder implemented within areceiver communication device.

As mentioned with respect to other embodiments, the use of an ALF canprovide any of a number of improvements in accordance with such videoprocessing, including an improvement on the objective quality measure bythe peak to signal noise ratio (PSNR) that comes from performing randomquantization noise removal. In addition, the subjective quality of asubsequently encoded video signal may be achieved from illuminationcompensation, which may be introduced in accordance with performingoffset processing and scaling processing (e.g., in accordance withapplying a gain) in accordance with ALF processing.

With respect to any video encoder architecture implemented to generatean output bitstream, it is noted that such architectures may beimplemented within any of a variety of communication devices. The outputbitstream may undergo additional processing including error correctioncode (ECC), forward error correction (FEC), etc. thereby generating amodified output bitstream having additional redundancy deal therein.Also, as may be understood with respect to such a digital signal, it mayundergo any appropriate processing in accordance with generating acontinuous time signal suitable for or appropriate for transmission viaa communication channel. That is to say, such a video encoderarchitecture may be implemented within a communication device operativeto perform transmission of one or more signals via one or morecommunication channels. Additional processing may be made on an outputbitstream generated by such a video encoder architecture therebygenerating a continuous time signal that may be launched into acommunication channel.

FIG. 7 is a diagram illustrating an embodiment 700 of intra-predictionprocessing. As can be seen with respect to this diagram, a current blockof video data (e.g., often times being square in shape and includinggenerally N×N pixels) undergoes processing to estimate the respectivepixels therein. Previously coded pixels located above and to the left ofthe current block are employed in accordance with such intra-prediction.From certain perspectives, an intra-prediction direction may be viewedas corresponding to a vector extending from a current pixel to areference pixel located above or to the left of the current pixel.Details of intra-prediction as applied to coding in accordance withH.264/AVC are specified within the corresponding standard (e.g.,International Telecommunication Union, ITU-T, TELECOMMUNICATIONSTANDARDIZATION SECTOR OF ITU, H.264 (March/2010), SERIES H: AUDIOVISUALAND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services—Coding ofmoving video, Advanced video coding for generic audiovisual services,Recommendation ITU-T H.264, also alternatively referred to asInternational Telecomm ISO/IEC 14496-10—MPEG-4 Part 10, AVC (AdvancedVideo Coding), H.264/MPEG-4 Part 10 or AVC (Advanced Video Coding), ITUH.264/MPEG4-AVC, or equivalent) that is incorporated by reference above.

The residual, which is the difference between the current pixel and thereference or prediction pixel, is that which gets encoded. As can beseen with respect to this diagram, intra-prediction operates usingpixels within a common frame (or picture). It is of course noted that agiven pixel may have different respective components associatedtherewith, and there may be different respective sets of samples foreach respective component.

FIG. 8 is a diagram illustrating an embodiment 800 of inter-predictionprocessing. In contradistinction to intra-prediction, inter-predictionis operative to identify a motion vector (e.g., an inter-predictiondirection) based on a current set of pixels within a current frame (orpicture) and one or more sets of reference or prediction pixels locatedwithin one or more other frames (or pictures) within a frame (orpicture) sequence. As can be seen, the motion vector extends from thecurrent frame (or picture) to another frame (or picture) within theframe (or picture) sequence. Inter-prediction may utilize sub-pixelinterpolation, such that a prediction pixel value corresponds to afunction of a plurality of pixels in a reference frame or picture.

A residual may be calculated in accordance with inter-predictionprocessing, though such a residual is different from the residualcalculated in accordance with intra-prediction processing. In accordancewith inter-prediction processing, the residual at each pixel againcorresponds to the difference between a current pixel and a predictedpixel value. However, in accordance with inter-prediction processing,the current pixel and the reference or prediction pixel are not locatedwithin the same frame (or picture). While this diagram showsinter-prediction as being employed with respect to one or more previousframes or pictures, it is also noted that alternative embodiments mayoperate using references corresponding to frames before and/or after acurrent frame. For example, in accordance with appropriate bufferingand/or memory management, a number of frames may be stored. Whenoperating on a given frame, references may be generated from otherframes that precede and/or follow that given frame.

Coupled with the CU, a basic unit may be employed for the predictionpartition mode, namely, the prediction unit, or PU. It is also notedthat the PU is defined only for the last depth CU, and its respectivesize is limited to that of the CU.

FIG. 9 and FIG. 10 are diagrams illustrating various embodiments 900 and1000, respectively, of video decoding architectures.

Generally speaking, such video decoding architectures operate on aninput bitstream. It is of course noted that such an input bitstream maybe generated from a signal that is received by a communication devicefrom a communication channel. Various operations may be performed on acontinuous time signal received from the communication channel,including digital sampling, demodulation, scaling, filtering, etc. suchas may be appropriate in accordance with generating the input bitstream.Moreover, certain embodiments, in which one or more types of errorcorrection code (ECC), forward error correction (FEC), etc. may beimplemented, may perform appropriate decoding in accordance with suchECC, FEC, etc. thereby generating the input bitstream. That is to say,in certain embodiments in which additional redundancy may have been madein accordance with generating a corresponding output bitstream (e.g.,such as may be launched from a transmitter communication device or fromthe transmitter portion of a transceiver communication device),appropriate processing may be performed in accordance with generatingthe input bitstream. Overall, such a video decoding architectures andlamented to process the input bitstream thereby generating an outputvideo signal corresponding to the original input video signal, asclosely as possible and perfectly in an ideal case, for use in beingoutput to one or more video display capable devices.

Referring to the embodiment 900 of FIG. 9, generally speaking, a decodersuch as an entropy decoder (e.g., which may be implemented in accordancewith CABAC, CAVLC, etc.) processes the input bitstream in accordancewith performing the complementary process of encoding as performedwithin a video encoder architecture. The input bitstream may be viewedas being, as closely as possible and perfectly in an ideal case, thecompressed output bitstream generated by a video encoder architecture.Of course, in a real-life application, it is possible that some errorsmay have been incurred in a signal transmitted via one or morecommunication links. The entropy decoder processes the input bitstreamand extracts the appropriate coefficients, such as the DCT coefficients(e.g., such as representing chroma, luma, etc. information) and providessuch coefficients to an inverse quantization and inverse transformblock. In the event that a DCT transform is employed, the inversequantization and inverse transform block may be implemented to performan inverse DCT (IDCT) operation. Subsequently, A/D blocking filter isimplemented to generate the respective frames and/or picturescorresponding to an output video signal. These frames and/or picturesmay be provided into a picture buffer, or a digital picture buffer (DPB)for use in performing other operations including motion compensation.Generally speaking, such motion compensation operations may be viewed ascorresponding to inter-prediction associated with video encoding. Also,intra-prediction may also be performed on the signal output from theinverse quantization and inverse transform block. Analogously as withrespect to video encoding, such a video decoder architecture may beimplemented to perform mode selection between performing it neitherintra-prediction nor inter-prediction, inter-prediction, orintra-prediction in accordance with decoding an input bitstream therebygenerating an output video signal.

Referring to the embodiment 1000 of FIG. 10, in certain optionalembodiments, one or more in-loop filters (e.g., implemented inaccordance with adaptive loop filter (ALF), sample adaptive offset (SAO)filter, and/or any other filter type) such as may be implemented inaccordance with video encoding as employed to generate an outputbitstream, a corresponding one or more in-loop filters may beimplemented within a video decoder architecture. In one embodiment, anappropriate implementation of one or more such in-loop filters is afterthe de-blocking filter.

FIG. 11 illustrates an embodiment 1100 of a video encoding architecture(with sample adaptive offset (SAO) implemented after de-blocking).Sample Adaptive Offset (SAO) (e.g., in accordance with a currentadoption in HEVC) may be employed for reducing any existent distortionbetween original pictures and reconstructed pictures caused byquantization. However, in accordance with such a proposal of SAO, suchSAO operations are performed after de-quantization and inverse transform(e.g., in accordance with FIG. 11), which is not a very efficientimplementation.

FIG. 12 illustrates an embodiment 1200 of a video encoding architecture(with a frequency-domain sample adaptive offset (FSAO) implemented afterinverse quantization). Herein, a novel implementation is proposed bywhich frequency domain SAO (FSAO) is implemented and may be carried outimmediately after dequantization (e.g., in accordance with FIG. 12). Afrequency-domain implemented SAO (FSAO) is relatively much more accuratesince it avoids any possible distribution caused by inverse transform.

As may be seen within this diagram, a frequency-domain implementedsample adaptive offset (FSAO) is instead implemented after inversequantization and before inverse transform in this alternative embodimentof a video encoder architecture.

With respect to implementing such frequency domain operations, spectralband replication (SBR) may be performed with respect to the videoinformation of a received signal. For example, with respect to areceived input bitstream (such as a video signal), a majority of energyor information included within that received input bitstream may bebelow a certain frequency. For example, in accordance with performingvideo encoding, particularly with respect to the transformingquantization operations performed, at least in part, in generating anoutput bitstream from an input video signal (e.g. which may berepresented as CUs, MBs, etc.), any energy or information above aparticular frequency is generally lost in accordance with compressionprocessing. That is to say, relatively high frequency energy orinformation included within the input video signal is generally lost inaccordance with video compression (e.g., such as in accordance withtransforming from the time domain to the frequency domain).

As such, and output bitstream, such as generated by a video encoder,will typically not include a great deal of energy or information above acertain frequency (e.g., a cutoff frequency, f_(c), corresponding tothat frequency above which energy or information is lost in accordancewith compression processing). As such, an output bitstream may notinclude all high-frequency energy or information therein. As such, ininput bitstream received by a video decoder, when reconstructing orgenerating an output video signal there from, will not have access tosuch high-frequency energy or information. For example, whentransmitting a video signal from a transmitter device to a receiverdevice (e.g., which may include any appropriate processing to generate asignal that comports with the communication channel, transmitting thecommunication channel compliance signal from the transmitter device tothe receiver device, and receiving the transmitted signal from thecommunication channel at the receiver device, etc.), such a receiverdevice will not have access to all high-frequency energy or informationthat may have been contained within an original or prior input videosignal.

However, in accordance with performing video decoding, such a device maybe implemented nonetheless to reconstruct such high-frequency energy orinformation based upon information included within the video signalreceived and/or the input bitstream corresponding thereto or derivedtherefrom.

For example, in accordance with performing video decoding processing,all of the energy or information within the signal may be identified asbeing below a certain frequency (e.g., a cutoff frequency, f_(c),corresponding to that frequency above which energy or information islost in accordance with compression processing). In accordance withreconstructing relatively high frequency information within a videosignal, spectral band replication (SBR) may be used to improve thecoding efficiency of such a video signal. While many exemplaryembodiments are described herein respect to video coding applications,it is also noted that such SBR may also be applied to audio signalsand/or other types of signals without departing from the scope andspirit of the invention.

For example, SBR may be viewed as being based on and using harmonicredundancy in the frequency domain of energy or information associatedwith the signal. Again, within the context of video coding, a majorityor all of the energy information will be located below a certainfrequency (e.g., a cutoff frequency, f_(c), corresponding to thatfrequency above which energy or information is lost in accordance withcompression processing). Moreover, in the context of video coding, it isalso noted that the use of such SBR may be made based upon the principlethat the perceptual capabilities of the human eye and human brain astending to analyze higher frequency information with relatively lessaccuracy. Analogously, with respect to audio coding, the psychoacousticpart of the human brain tends to analyze higher frequency audioinformation with relatively less accuracy as well. As such, harmonicphenomenon or phenomena associated with such an SBR process need only beaccurate within a perceptual sense based on human perceptual capability,and it need not be technically or mathematically exact (because of theincapability of human perception to discern such high-frequencyinformation with a high degree of accuracy).

FIG. 13 illustrates an embodiment 1300 of principles of spectral bandreplication (SBR)—transposition (a) and reconstruction (b). With respectto this diagram it can be seen on the left-hand side of the diagram thata relatively lower frequency portion of energy or information mayundergo transposition or replication to a higher frequency portion ofthe frequency spectrum. On the right-hand side of the diagram, thatreplicated energy or information may further undergo appropriate scaling(e.g., shown as downward scaling or attenuation in this embodiment) toeffectuate a relatively smooth transition between the original energy orinformation that had undergone transposition replication and thereplicated and scaled energy or information. From certain perspectives,an energy profile (e.g., energy as a function of frequency)corresponding to at least one portion of a video signal undergoesreplication (or replication and scaling) to generate another portion ofthe video signal (e.g., such as within a different frequency spectralportion). While many of the embodiments described herein includereplication (or replication and scaling) from a relatively lowerfrequency portion to a relatively higher frequency portion. Of course,such replication (or replication and scaling) may alternatively beperformed from a relatively higher frequency portion to a relativelylower frequency portion in any alternative embodiment.

Generally speaking, many signals typically exhibit relatively lessenergy or information at higher frequencies. For example, with respectto video signals, a majority of the energy or information is included atrelatively lower frequencies. This is analogous with audio signals inwhich a majority of the energy or information is included at relativelylower frequencies. Moreover, given the perceptual limitations of thehuman eye, year, and/or brain, a significant amount of information thatthese relatively higher frequencies is typically not needed and/or doesnot provide a significant improvement in perceptual quality. Within thisdiagram as well as others, the amount of energy or information (as afunction of frequency) is generally monotonically decreasing asfrequency increases. However, as may be seen, there may be someinstances in which there are ripples within this general trend ofmonotonic decrease.

Generally speaking, inside of an image (e.g., an image of a videosignal) there may be a lot of edge information (e.g., relativelyhigh-frequency information), and if analyze in the frequency domain, thecorresponding frequency spectrum ranges from DC to relativelyhigh-frequency. Usually, the DC and relatively low frequency componentshave relatively higher magnitude than the higher frequency componentswithin the total amount of energy or information of the signal.

If the high-frequency components are removed or heavily quantized (e.g.,such as in accordance with transformation and quantization as performedin accordance with video encoding), and if only the relatively lowerfrequency and/or middle frequency portions of a signal are employed(e.g., such as transmitted within a signal from a transmitter device toreceiver device), then the sharp edge information will be unfortunatelysmoothed out.

The use of SBR may be employed in accordance with video decoding toreplicate such removed higher frequency components by transposingharmonics from the received signal (e.g., that includes relatively lowerand/or middle frequency portions of the signal). As such, thisrelatively higher frequency information may be reconstructed inaccordance of video decoding. Such SBR related processing may beperformed on any of a number of bases including a CU basis, an MB basis,etc. without departing from the scope and spirit of the invention.

FIG. 14, FIG. 15, FIG. 16, and FIG. 17 illustrate various embodiments ofspectral band replication (SBR).

Referring to embodiment 1400 of FIG. 14, as may be seen with respect tothis diagram, the entirety of energy or information below a certainfrequency (e.g., a cutoff frequency, f_(c), corresponding to thatfrequency above which energy or information is lost in accordance withcompression processing in accordance with video processing) isreplicated to a higher frequency. In addition, this replicated versionof the energy or information is scaled (e.g., shown as downward scalingor attenuation in this embodiment) to effectuate a relatively smoothtransition between the original energy or information that had undergonetransposition replication and the replicated and scaled energy orinformation.

Referring to embodiment 1500 of FIG. 15, as may be seen with respect tothis diagram, the entirety of energy or information below a certainfrequency (e.g., a cutoff frequency, f_(c), corresponding to thatfrequency above which energy or information is lost in accordance withcompression processing in accordance with video processing) isreplicated to a number of higher frequency bands as may be seen withsuspected to be constructed energy or information 1, reconstructedenergy or information 2, and so on as many as may be desired in aparticular embodiment.

Referring to embodiment 1600 of FIG. 16, as may be seen with respect tothis diagram, a portion or sub-portion of energy or information within acertain frequency band (e.g., above a first frequency, f₁, and below asecond frequency, f₂) is replicated to a number (e.g., which may be oneor more) of higher frequency bands. That is to say, as few as onereconstruction of this portion or sub-portion of energy or informationmay be replicated in one or more higher frequency spectral bands. Ifdesired, multiple respective reconstructions of this portion orsub-portion of energy or information may be replicated in multiplerespective higher frequency spectral bands.

Referring to embodiment 1700 of FIG. 17, as may be seen with respect tothis diagram, more than one respective portion or sub-portion of energyor information within one or more frequency bands (e.g., a first bandcorresponding to energy or information above a first frequency, f₁, andbelow a second frequency, f₂—a second band corresponding to energy orinformation above the second frequency, f₂, and below a third frequency,such as f_(c), etc.) is replicated to a number (e.g., which may be oneor more) of higher frequency bands. That is to say, as few as onerespective reconstruction of each of these respective portions orsub-portions of energy or information may be replicated in one or morehigher frequency spectral bands. If desired, multiple respectivereconstructions of these respective portions or sub-portions of energyor information may be replicated in multiple respective higher frequencyspectral bands.

If the high-frequency components are heavily quantized but notcompletely removed (which means some high-frequency components areremained after quantization but have large quantization distortion),they can be used as pilots during the SBR process. That is to say, thereplicated high frequency bands has to be consistent with the scale andsign of the remaining high-frequency components.

Generally speaking, it is noted that any of a number of differentrespective portions or sub-portions of energy or information within oneor more frequency bands may be used in accordance with SBR processing.For example, in accordance with video decoding, multiple respectiveportions or sub-portions of energy or information within one or morefrequency bands may be employed in accordance with performing SBRprocessing thereby generating higher frequency energy or informationwithin one or more higher frequency spectral bands.

In alternative embodiments, it is noted that coordination may beeffectuated between a transmitter device and receiver device such thatthe transmitter device provides information to the receiver device(e.g., either within a separate control channel, or embedded within thesignal transmitted that includes the video information or video signal,etc.) providing certain information to the receiver device for directingoperation of the receiver device. For example, a receiver device couldbe configured or implemented to perform and identify any of a number ofoperations independently without coordination with the transmitterdevice (e.g., perhaps other than receiving a video signal there from).Alternatively, receiver device could be configured are implemented toreceive particular information or direction from a transceiver device todirect the manner by which the receiver device should perform decodingand other video processing of a received video signal. Such informationmay include the cutoff frequency associated with transform andquantization operations in accordance with video encoding associatedwith a video signal that is transmitted from a transmitter device to areceiver device. In addition, further instruction may be provided from atransmitter device to receiver device including a number of times thatone or more portions or sub-portions of the video signal should bereplicated and/or scaled, one or more portions or sub-portions of thevideo signal that should be replicated or scaled (including, asapplicable, upper and lower frequency bounds associated with any suchsub-portions), one or more amounts or types of scaling to be applied toone or more portions or sub-portions of the video signal, a number oftimes that one or more portions or sub-portions of the video signalshould be replicated and/or scaled, etc. As may be understood, any of anumber of different types of information, including control information(e.g., magnitude(s) as may be employed for scaling(s), frequencyrange(s) for reconstruction of a high-frequency spectral envelope,etc.), may be provided from my transmitter device to receiver device fordirecting operation of the receiver device.

FIG. 18, FIG. 19, FIG. 20A, and FIG. 20B illustrate various embodimentsof methods for performing video processing (e.g., within one or moredevices)

Referring to method 1800 of FIG. 18, the method 1800 begins by selectinga first at least a portion of video information from a first videosignal, as shown in a block 1810.

The method 1800 continues by replicating at least one characteristic ofthe first at least a portion of video information to generate a secondat least a portion of video information within a relatively higherfrequency range than the first at least a portion of video information,as shown in a block 1820. Such a characteristic may correspond to anenergy profile of at least one portion of the first video signal (e.g.,energy as a function of frequency, such as in accordance with FIGS.13-17, and such description referenced in the written descriptionassociated with other diagrams.

The method 1800 then operates by combining the first at least a portionof video information and the second at least a portion of videoinformation to generate a second video signal, as shown in a block 1830.The video processing operations as described herein may be performed inany of a variety of bases, including on a block by block basis (or viaany other partitioning of one or more video signals).

In certain embodiments, the method 1800 may also continue by outputtingthe second video signal or a signal corresponding to the second videosignal via an output of a communication device, as shown in a block1840.

Referring to method 1900 of FIG. 19, the method 1900 begins by selectinga first at least a portion of video information from a first videosignal, as shown in a block 1910.

The method 1900 continues by replicating and scaling the first at leasta portion of video information to generate a second at least a portionof video information within a relatively higher frequency range than thefirst at least a portion of video information, as shown in a block 1920.

The method 1900 then operates by combining the first at least a portionof video information and the second at least a portion of videoinformation to generate a second video signal, as shown in a block 1930.The video processing operations as described herein may be performed inany of a variety of bases, including on a block by block basis (or viaany other partitioning of one or more video signals).

In certain embodiments, the method 1900 may also continue by outputtingthe second video signal or a signal corresponding to the second videosignal via an output of a communication device, as shown in a block1940.

Referring to method 2000 of FIG. 20A, the method 2000 begins byemploying a first subset of the first at least a portion of videoinformation of a first video signal to generate a first subset of thesecond at least a portion of video information within a first relativelyhigher frequency range than the first at least a portion of videoinformation in accordance with generating a second video signal, asshown in a block 2010.

The method 2000 continues by employing a second subset of the first atleast a portion of video information of the first video signal togenerate a second subset of the second at least a portion of videoinformation within a second relatively higher frequency range than thefirst at least a portion of video information in accordance withgenerating a second video signal, as shown in a block 2020.

Referring to method 2001 of FIG. 20B, the method 2001 begins byemploying a first at least a portion of video information of a firstvideo signal to generate a first subset of a second at least a portionof video information within a first relatively higher frequency rangethan the first at least a portion of video information in accordancewith generating a second video signal, as shown in a block 2011.

The method 2001 then operates by employing the first at least a portionof video information of a first video signal to generate a second subsetof the second at least a portion of video information within a secondrelatively higher frequency range than the first at least a portion ofvideo information in accordance with generating a second video signal,as shown in a block 2021.

It is also noted that the various operations and functions as describedwith respect to various methods herein may be performed within acommunication device, such as using a baseband processing module and/ora processing module implemented therein and/or other component(s)therein.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “operably coupled to”, “coupled to”, and/or “coupling” includesdirect coupling between items and/or indirect coupling between items viaan intervening item (e.g., an item includes, but is not limited to, acomponent, an element, a circuit, and/or a module) where, for indirectcoupling, the intervening item does not modify the information of asignal but may adjust its current level, voltage level, and/or powerlevel. As may further be used herein, inferred coupling (i.e., where oneelement is coupled to another element by inference) includes direct andindirect coupling between two items in the same manner as “coupled to”.As may even further be used herein, the term “operable to” or “operablycoupled to” indicates that an item includes one or more of powerconnections, input(s), output(s), etc., to perform, when activated, oneor more its corresponding functions and may further include inferredcoupling to one or more other items. As may still further be usedherein, the term “associated with”, includes direct and/or indirectcoupling of separate items and/or one item being embedded within anotheritem. As may be used herein, the term “compares favorably”, indicatesthat a comparison between two or more items, signals, etc., provides adesired relationship. For example, when the desired relationship is thatsignal 1 has a greater magnitude than signal 2, a favorable comparisonmay be achieved when the magnitude of signal 1 is greater than that ofsignal 2 or when the magnitude of signal 2 is less than that of signal1.

As may also be used herein, the terms “processing module”, “module”,“processing circuit”, and/or “processing unit” (e.g., including variousmodules and/or circuitries such as may be operative, implemented, and/orfor encoding, for decoding, for baseband processing, etc.) may be asingle processing device or a plurality of processing devices. Such aprocessing device may be a microprocessor, micro-controller, digitalsignal processor, microcomputer, central processing unit, fieldprogrammable gate array, programmable logic device, state machine, logiccircuitry, analog circuitry, digital circuitry, and/or any device thatmanipulates signals (analog and/or digital) based on hard coding of thecircuitry and/or operational instructions. The processing module,module, processing circuit, and/or processing unit may have anassociated memory and/or an integrated memory element, which may be asingle memory device, a plurality of memory devices, and/or embeddedcircuitry of the processing module, module, processing circuit, and/orprocessing unit. Such a memory device may be a read-only memory (ROM),random access memory (RAM), volatile memory, non-volatile memory, staticmemory, dynamic memory, flash memory, cache memory, and/or any devicethat stores digital information. Note that if the processing module,module, processing circuit, and/or processing unit includes more thanone processing device, the processing devices may be centrally located(e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that if the processing module, module, processing circuit,and/or processing unit implements one or more of its functions via astate machine, analog circuitry, digital circuitry, and/or logiccircuitry, the memory and/or memory element storing the correspondingoperational instructions may be embedded within, or external to, thecircuitry comprising the state machine, analog circuitry, digitalcircuitry, and/or logic circuitry. Still further note that, the memoryelement may store, and the processing module, module, processingcircuit, and/or processing unit executes, hard coded and/or operationalinstructions corresponding to at least some of the steps and/orfunctions illustrated in one or more of the Figures. Such a memorydevice or memory element can be included in an article of manufacture.

The present invention has been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claimed invention. Further, theboundaries of these functional building blocks have been arbitrarilydefined for convenience of description. Alternate boundaries could bedefined as long as the certain significant functions are appropriatelyperformed. Similarly, flow diagram blocks may also have been arbitrarilydefined herein to illustrate certain significant functionality. To theextent used, the flow diagram block boundaries and sequence could havebeen defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claimed invention. One of average skill in the artwill also recognize that the functional building blocks, and otherillustrative blocks, modules and components herein, can be implementedas illustrated or by discrete components, application specificintegrated circuits, processors executing appropriate software and thelike or any combination thereof.

The present invention may have also been described, at least in part, interms of one or more embodiments. An embodiment of the present inventionis used herein to illustrate the present invention, an aspect thereof, afeature thereof, a concept thereof, and/or an example thereof. Aphysical embodiment of an apparatus, an article of manufacture, amachine, and/or of a process that embodies the present invention mayinclude one or more of the aspects, features, concepts, examples, etc.described with reference to one or more of the embodiments discussedherein. Further, from figure to figure, the embodiments may incorporatethe same or similarly named functions, steps, modules, etc. that may usethe same or different reference numbers and, as such, the functions,steps, modules, etc. may be the same or similar functions, steps,modules, etc. or different ones.

Unless specifically stated to the contra, signals to, from, and/orbetween elements in a figure of any of the figures presented herein maybe analog or digital, continuous time or discrete time, and single-endedor differential. For instance, if a signal path is shown as asingle-ended path, it also represents a differential signal path.Similarly, if a signal path is shown as a differential path, it alsorepresents a single-ended signal path. While one or more particulararchitectures are described herein, other architectures can likewise beimplemented that use one or more data buses not expressly shown, directconnectivity between elements, and/or indirect coupling between otherelements as recognized by one of average skill in the art.

The term “module” is used in the description of the various embodimentsof the present invention. A module includes a functional block that isimplemented via hardware to perform one or module functions such as theprocessing of one or more input signals to produce one or more outputsignals. The hardware that implements the module may itself operate inconjunction software, and/or firmware. As used herein, a module maycontain one or more sub-modules that themselves are modules.

While particular combinations of various functions and features of thepresent invention have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent invention is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. An apparatus, comprising: a video processor to:select a first at least a portion of video information from a firstvideo signal; replicate and scale the first at least a portion of videoinformation to generate a second at least a portion of video informationwithin a relatively higher frequency range than the first at least aportion of video information; and combine the first at least a portionof video information and the second at least a portion of videoinformation in accordance with processing the first video signal togenerate a second video signal.
 2. The apparatus of claim 1, wherein:the video processor to employ a first subset of the first at least aportion of video information to generate a first subset of the second atleast a portion of video information within a first relatively higherfrequency range than the first at least a portion of video information;and the video processor to employ a second subset of the first at leasta portion of video information to generate a second subset of the secondat least a portion of video information within a second relativelyhigher frequency range than the first at least a portion of videoinformation.
 3. The apparatus of claim 1, wherein: the video processorto employ the first at least a portion of video information to generatea first subset of the second at least a portion of video informationwithin a first relatively higher frequency range than the first at leasta portion of video information and a second subset of the second atleast a portion of video information within a second relatively higherfrequency range than the first at least a portion of video information.4. The apparatus of claim 1, wherein: the apparatus being a videodecoder.
 5. The apparatus of claim 1, wherein: the apparatus being acommunication device operative within at least one of a satellitecommunication system, a wireless communication system, a wiredcommunication system, a fiber-optic communication system, and a mobilecommunication system.
 6. An apparatus, comprising: a video processor to:select a first at least a portion of video information from a firstvideo signal; replicate at least one characteristic of the first atleast a portion of video information to generate a second at least aportion of video information within a relatively higher frequency rangethan the first at least a portion of video information; and combine thefirst at least a portion of video information and the second at least aportion of video information to generate a second video signal.
 7. Theapparatus of claim 6, wherein: the video processor to replicate andscale the first at least a portion of video information to generate thesecond at least a portion of video information.
 8. The apparatus ofclaim 6, wherein: the at least one characteristic of the first at leasta portion of video information corresponding to an energy profile as afunction of frequency.
 9. The apparatus of claim 6, wherein: the videoprocessor to employ a first subset of the first at least a portion ofvideo information to generate a first subset of the second at least aportion of video information within a first relatively higher frequencyrange than the first at least a portion of video information; and thevideo processor to employ a second subset of the first at least aportion of video information to generate a second subset of the secondat least a portion of video information within a second relativelyhigher frequency range than the first at least a portion of videoinformation.
 10. The apparatus of claim 6, wherein: the video processorto employ the first at least a portion of video information to generatea first subset of the second at least a portion of video informationwithin a first relatively higher frequency range than the first at leasta portion of video information and a second subset of the second atleast a portion of video information within a second relatively higherfrequency range than the first at least a portion of video information.11. The apparatus of claim 6, wherein: the video processor to operate ona block by block basis in accordance with processing the first videosignal to generate the second video signal.
 12. The apparatus of claim6, wherein: the apparatus being a video decoder.
 13. The apparatus ofclaim 6, wherein: the apparatus being a communication device operativewithin at least one of a satellite communication system, a wirelesscommunication system, a wired communication system, a fiber-opticcommunication system, and a mobile communication system.
 14. A methodfor operating a communication device, the method comprising: selecting afirst at least a portion of video information from a first video signal;replicating at least one characteristic of the first at least a portionof video information to generate a second at least a portion of videoinformation within a relatively higher frequency range than the first atleast a portion of video information; combining the first at least aportion of video information and the second at least a portion of videoinformation to generate a second video signal; and via an output of thecommunication device, outputting the second video signal or a signalcorresponding to the second video signal.
 15. The method of claim 14,further comprising: replicating and scaling the first at least a portionof video information to generate the second at least a portion of videoinformation.
 16. The method of claim 14, wherein: the at least onecharacteristic of the first at least a portion of video informationcorresponding to an energy profile as a function of frequency.
 17. Themethod of claim 14, further comprising: employing a first subset of thefirst at least a portion of video information to generate a first subsetof the second at least a portion of video information within a firstrelatively higher frequency range than the first at least a portion ofvideo information; and employing a second subset of the first at least aportion of video information to generate a second subset of the secondat least a portion of video information within a second relativelyhigher frequency range than the first at least a portion of videoinformation.
 18. The method of claim 14, further comprising: employingthe first at least a portion of video information to generate a firstsubset of the second at least a portion of video information within afirst relatively higher frequency range than the first at least aportion of video information and a second subset of the second at leasta portion of video information within a second relatively higherfrequency range than the first at least a portion of video information.19. The method of claim 14, wherein: processing the first video signalon a block by block basis in accordance generating the second videosignal.
 20. The method of claim 14, wherein: the communication deviceoperative within at least one of a satellite communication system, awireless communication system, a wired communication system, afiber-optic communication system, and a mobile communication system.